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DELIVERY SYSTEM AND METHOD OF REAL-TIME MULTIMEDIA STREAMS 



BACKGROUND OF THE INVENTION 



1. b'ield of the Invention 

The prcscnL invention relates to delivery techniques of 
real-time data streams and in particular to system and method 
b of delivering real-time data streams over a network. 

2. Description o£ the Related Art 

In general, a video transmission system may be divided 
into three main types of inf restructure: Television 
broadcasting such as Broadcasting Satellite (BS) or 

10 Communication Satellite (CS) ; Cable Television (CATV) 

broadcasting; and Internet broadcasting. A recent shift from 
analog-base to digital-base makes a wide variety of hitherto 
impossible services possible. In the field of Television 
broadcasting, for example, the following services become 

lb feasible: 

• Several hundred high-quality channels tor providing 
programs ; 

• EPC (Electronic Programming Guide) ; 

• Program reservations and watching using a home server; 

20 • Program-synchronization broadcasting such that, lor example, 
a www homepage i s synchronized to a program, which i s reali zed 
in analog broadcasting systems such as ADAMS (TV-Asahi Data 
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and Multimedia Service) or BiLCasL; and 

• Audience participation live program Lhrough Interactive 
(bi direct tonal) broadcast inq . 

In addition, ihere has been proposed a system such as 
5 Real Video G2, which is capable of presenting single content 
formed by combi ning content materials cuch as stored video data, 
audio data, and/ or HTML document data. This conventional system 
allows the combined content materials to be synchronous' ly 
displayed according to a predetermi ned scenario, achieving more 
10 widely various represent* Lions , compared with conventionally 
formed content - 

Under these circumstances, a technique of multicasting 
live audio and video data, or real-time multimedia data, to 
a plurality of participants has received widespread attention, 
lb and therefore various multimedia content multicasting 

techniques have been proposed (for example, see US Patent No. 
US 6, 'Jbl, 467 Bl) . 

On the other hand, a 1 i ve-vi deo transmi s s i on system whi ch 
transmits live video data from a transmitter to a receiver has 
20 been described in Japanese Patent Application Unexamined 
Publication No. P2001-45445A. More specifically, this 
live-video transmission system attaches picture ID and indexes 
to the live video data, allowing a user to select desired pictures 
of Lhe live content . This conventional live-video transmission 
25 system is based on the unicast technology. 

In the multicast delivery ol real-time content streams, 
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however, time-shifting operations of the real-time content, 
including pause, revere, fas t-lorward, and Lime-shift 
repl aying during recording, are not possible. Tn other words, 
iL is impossible to play buck awhile-preceding live video 
5 pictures, Taking an electronic conference using 

the multicasting Lechnology as an example, when a participant 
ia late for the electronic conference, the participant cannot 
reference previous presentation documents and therefore may not 
understand well a discussion that is currently made in the 
10 electronic conference. 

SUMMARY OF THE INVENTION 

An object o£ the present Invention is to provide a system 
and method for delivering real-Lime data streams allowing 
time-shifting operations such as pause, reverse, fast-forward, 
15 and time-shift replaying. 

According to the present invent 1 on, a transmi tter delivers 
real-Lime multimedia data in multicast to the receivers while 
k tori ng the real -time mul timed in data into n first memory. When 
havi ng received a time sh 1 ft transition command from a receiver, 
20 Lhe transmitter reads time-shifted multimedia data from the 
first memory depending on the time-shift transition command. 
The timo-ehif ted multimedia data is then transmitted in unicast 
Lo the receiver which originated the Lime- shirt transition 
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command. The receiver receives the real-time multimedia data 
in multicast from Lhe transmitter before transmitting the 
time-shift transition comrciand and receives the 
time-shifted multimedia data in unicast from Lhc transmitter 
5 afler Iransmitting the time-shift transition command. 

The transmitter preferably adds a time stamp to each 
transmission block, size of Lhe real-time mul Limcdia daLa, stores 
Lhe real-time multimedia data with time stamps into the 
first memory, and delivers the real-time multimedia data with 

10 time stamps to the receivers. 

. The receiver preferably creates a thumbnail picture from 
the real-time multimedia data received from the transmitter 
each time an amount of real-Lime multimedia daLa per unit Lime 
exceeds a predetermined level, and stores thumbnail pictures 

15 with corresponding time stamps into a second memory so as to 
designate a desired Lhumbnail picture/ allowing a desired 
location oi the real-time multimedia data Lo be designated. 
In addition,, when a time-shift request occurs, the receiver 
creates a time-shift transition command based on the thumbnail 

20 pictures with the corresponding time stamps stored in the 
second memory, and transmits Lhe time-shift transition command 

Lo the transmitter so as to receive time-shifted multimedia 
data from the transmitter in unicast. 

The time-shift transition command may be one: of a replay 
2b start location designation command, a pause command, a reverse 
command, a slow-rcplay command, and a fast -forward command. 
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According to another embodiment, each of the receivers 
stores the real-Lime multimedia data received from the 
transmitter into a third memory and, when a times-shift request 
occurs, reads time-shifted multimedia data from the third memory 
5 depending on the time-shift request. 

According to anther aspect of the present invention, when 
having. received a start request command from a receiver, the 
transmitter delivers real-time multimedia data. in multicast 
to the receiver while storing the real-time multimedia data 
1 0 into a first memory . When ha vi ng rece i ved a time-shift transit i on 
command from the receiver, the transmitter reads 
time-shifted multimedia data from the first memory depending 
on the lime-shift transition command, to transmit the 
Lime-shifted multimedia data in unicasL to the receiver; which 
15 originated the time-shift transition command. When having 
received a termination request command from the receiver,, the 
transmitter terminates multimedia data delivery to the 
receiver. 

The transmitter may include: an input section for 
20 inputting real-time multimedia data; a multicast transmit: icr 
for transmitting the real-lime multimedia data to each of the 
receivers; a first unicasL transceiver for receiving a command 
from a receiver and transmitting a response to the receiver; 
a command analyzer tor analyzing a command received from the 
25 receiver to determine a type of the received command; a 

first memory for storing the real-time multimedia data; and 
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a first controller controlling the multicast transmitter, the 
unicast transceiver and the first memory, such that the 
real time multimedia data is delivered in multicast to each 
oE the receivers while storing the real-time multimedia data 
5 into the f i r s I;, memory ; wherein, when having received a lime-shit t 
transition command from a receiver, time-shifted multimedia 
data is read from the first memory depending on the time-shift 
trans i Lion command and is Lrdusitii Lied in unicast Lo the receiver 
which originated the time-shift transition command. 

10 'The receiver may include: a multicast receiver for 

receiving the real-time mul ti media data from the transmitter; 
a second unicast transceiver for transmitting a command to the 
transmitter and receiving a response to the command from the 
transmitter; and a second controller controlling such that the 

15 real- time multimedia data is received in multicast from the 
transmitter before transmitting the time-shift transition 
command, and the time-shifted multimedia data is received in 
urn cast irom Lhe transmit ter after transmitting the time-shift 
transition command . 

20 As described above, the real-time multimedia stream 

delivery system according to the present invention is provided 
with a muliicasL-unicast changeover mechanism for changing 
between real-time multimedia data delivery and 
time-shi fted multimedia data delivery, which allows 

25 time-shifted replay in real-time multimedia streaming delivery. 
In other words, the real-time multimedia data is transmitted 
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inmulticast and the time-shif ted mult imedia data is transmitted 
on demand to a receiver in unicast . Accordingly, each receiver 
having relatively small amount of memory capacity can provide 
time-shifted replay without storing real-time multimedia data . 
5 Further, when Lhe receiver creates thumbnail pictures 

i'rom the received real-time multimedia data, time stamps 
attached to the thumbnail pictures are stored into the storage 
devr ce. Accordingly, by just designating and sending a desired 
thumbnail picture to the transmitter, a user-desired scene of 
10 real-time multimedia data can be easily replayed. 

BR IKK INSCRIPTION OK THE DRAWINGS 

Fig. 1 is a block diagram showing a real-time si: ream 
delivery system according to a first embodiment of the present 
invention; 

15 Fig. 2 is u block diagram showing a functional structure 

of a controller in the real-time stream delivery system according 

to the first embodiment ; 

Fig. 3 is a f .1 ow chart showing a transmission control 
operation o P a I ransmi lie r in the real-time stream delivery 
20 system according to the first embodiment; 
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Fig . 4 is a 1 low char L showing a recopLion control operation 
ol a receiver in the real-time stream delivery system according 
to the first embodiment; 

fc'iq. 5 is a diagram showing a UDP format used in multicast 

5 transmission; 

Fig. 6 is a diagram showing a TCP format, used in uciicast 
transmission; 

h'iq. 7 is a diaqram showing the format of a command; 

Fig. 8 is a diagram showing the time-varying amount 
10 of multimedia data for explanation of creating thumbnail images 
in the lirst embodiment; 

Fig. 9 is a diagram showing a sequence of commuriicaLion 
when a start request command is sent from the receiver to the 
transmitter in the first embodiment; 

lb iig. 10 is a diagram showing a sequence of communication 

when a time-shift transition command 1s sent from the receiver 
to the transmitter in the first embodiment.; 

Fig. 11 is a diagram showing a sequence of communication 
when a termination request command is sent from the receiver 
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to the transmitter in the first embodiment; 

Fig* 12 is a block diagram showing a real-time stream 
delivery system according to a second embodiment ot the present 
invention; and 

Fig. 13 a block diagram showing a real- Lime a L ream 
delivery system according to a third embodiment of the present 
invent i on . 



As shown in Fig. 1, it is assumed for simplicity Lhal 
a delivery system of real-time multimedia streams according 
to a first embodiment. oL the present invention is composed of 
a transmitter 1U and a receiver 12, which arc connected by 
a network 11. Needless to say, a plurality of receivers 12 may 
be connected to the network 11.. The network 11 may be a 
communication network such as the Internet or Intranet. As 
described later, the transmitter 10 delivers 
real-time multimedia data such as live video data to the 
receiver (s) 12 in multicast through the network. When a 
Lime-shift replaying request occurs, the t ransmi LLe c 10 
delivers requested t i/mc-shi f ted multimedia data in unicast 
through the network 11 to the receiver 12 which originates that 
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request. The detailed structures and operations of the 
transmitter 10 and Lhe receiver 12 will be described hereinafter. 
Transmitter 

in the transmitter 10, an Input section 101 inputs 
5 real-time multimedia data including still pictures, moving 
pictures, audio data, and/or text data from an input device 
such as sLill/vidco camera, microphone and the likes. Here, 

Lhe multimedia data may be digital or analog. 

A data processing section 102 receives the multimedia 

10 data from the input section 1.01 and performs data processing 
such as anaiog-to-diyital conversion, format transform, and 
time stamp addition. The time stamp addition is to add time 
i n f Donation (Lime stamp) Lo Lhe rriul t Imedia daLa For each packet 
size. The multimedia data wiLh time a Lamps added lor each packet 

15 size are output to a controller 10-1 and stored in a storage 
device 103, which may be a memory device such as a hard disk 
drive (HDD) . 

The controller 104 inputs the output of the data process i ng 
section 102 and the output of the storage device 103 and selects 

20 one of these outputs depending on a command analytical result 
input Led from a command analyzer 105. The command analyzer 
10b analyzes a command received by a command receiver 108 to 
output its analytical result to the control 1 er 1 04 and a command 
response section 107. Depending on Lhe analytical result, the 

25 controller 104 selects one of the output of the data processing 
section 102 and Lhe ompul of the storage device 103 to output 
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selected multimedia data with time stamps to a transmission 
data processor 106. Detailed structure and operation of the 
controller 104 will be described later. 

The transmission data processor 106 assembles data packcLs 
b from the multimedia data with time stamps and, if necessary, 
encrypt 3 the payloads thereof. The data packets are outputted 
lo a selected one of a multicast transmitter 111 and a unicast 

transmitter 112 of a transmission section 109. 

The command response section 107 creates a command 
10 response packet according to the analytical result inputted 
from the command analyzer 10b and outputs it to the unicast 
transmitter 112. 

The command receiver 108 receives * command packet from 
a certain receiver 12 through a unicast receiver 113 of a 
15 reception section 110 and outputs its command to the command 
analyzer 10b. In this embodiment, d multicast receiver 114 
of the reception section 110 is not used. 
Recei ver 

Tn the receiver 1? , the receiver 12 is provided with a 
20 reception section lib including a mu.ltj.cast receiver 117 and 
a unicast receiver 118 arid a transmission section 116 including 
a unicast transmitter 119 and a multicast transmitter 120. 
The multicast receiver 117 receives real-time multimedia data 
from the multicast transmitter 111 of the transmitter '10. The 
255 unicast receiver 118 receive?:? time-shifted multimedia data or 
time-shift control data from the unicast transmitter 112 ol 
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the transmitter 10. In this embodiment , the multicast 
transmi t.tfir 120 is not used. 

A received data analyzer 121 analyzes packets received 
trom the transmitter 10 through Lhe unicust receiver 118 to 
5 determine whether a received packet is a data packet or a command 
response packet . When the da La packeL is received, the received 
data analyzer 121 outputs it as time-shifted multimedia data 
to a data reception selector 123. When the command response 
packet is received, the received data analyzer 121 outputs it 
10 to a controller 124. 

A command transmitter 122 inputs a command from the 
controller 124 to assemble a command packet , The command packet 
is outputted to the transmitter 10 through the uiiic**L 
transmitter 119 - 

lb The data reception selector 123 selects one of the 

real-time multimedia data received from the multicast receive! 
117 and the time-shifted multimedia data received from Lhe 
recei ved data analyzer 121 dependi n 9 on an i nstruction inputted 
from the controller 124. A selected multimedia data is 

20 outputted to a reception data processor 125. 

The controller 124 issues a command to a command 
transmitter 122, instructs the data reception selector 123 to 
perform received data selection as described above, and controls 
reception flag (TRUE/FALSE) and reception mode 

25 (real-Lime/ time-shift) as described later. 

The reception data processor 125 performs digital -to 
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analog conversion and format transform of received multimedia 
data, analyzes the received multimedia data and its attached 
time stamps, and creates thumbnail pictures depending uii Lhc 
amount of multimedia data per unit time, which will bo described 
5 later. 

a storage device 126 stores thumbnail pictures created 
by Lhc reception data processor 12 b together with Lime slumps. 
The storage device 126 may be a memory device such as a hard 
disk drive (HDD) , An output section 127 inputs mul ti media data 
10 from Lhc reception data processor 125 and the thumbnail pictures 
from the storage device 126 and outputs and, as necessary, 
combines them to an external monitor (not shown) . 
Multicast and Unicast 

Multicast transmission and Unicast transmission are 
15 defined as standards in RFC (Request For Comments) issued by 
IETF (Internet Engineering Task Force) . 

Multicast transmission is defined in RFC / 68 : User Datagram 
Protocol (UDP), in which UDP is used to transfer data. UDP 
has a relati vel y simp! e Format composed of : Source/Destination 
20 IP address fields; Protocol field (here, protocol number "17" 
is assigned)/ Packet length field; and Delta field (see Fig. 
5) . 

Unicast transmission is defined in RFC7 93: Transmission 
Control Protocol (TCP) , in which TCP ia used to transfer data. 
25 TCL> has a format composed of Source port field, Destination 
port Held, Sequence number field (indicating the location of 
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transmitted data), Acknowledgement number field (indicating 

the Sequence number oL next data Lo be transmitted) , Data offset 
field (indicating the start location of data) , Reserved field, 
Control tlag rield, window iieid (indicating the window size), 
5 Checksum field, Urgent pointer field, Options field, Padding 
field, and Data field (see Fig. 6). In Control flag field, 
ORG flag set Lo 1 indicates that Lhe data field includes urgent 
data, ACK flag set to 1 indicates that the acknowledgement field 
is valid (ACK flag is normally set to 1 other than connection 

10 establishment), PSH (Push) flag set to 1 indicates that buffering 
is not performed, RST (Reset) flag set to 1 indicates that a 
connection Is forced to be disconnected, SYN (Synchronize) flag 
set Lo 1 indicates thai a connection tries Lo be established, 
and FIN (Finish flag set to 1 indicates that the connection 

15 tries to be disconnected. The urgent pointer field indicates 
the length of urgent data included in data when URG flay is 
set to 1. 

Com mands 

A command is composed of a command field and an option 
20 field as shown in Fig. 7. The command field deiines a command 
depending on what value is written therein. For example, a 
101 value indicates Repl ay command, a 103 indicates Pause command, 
a 104 indicates Fast-Forward command/ a 105 indicates Reverse 
command, and a 10 6 indicates Thumbnail-based Replay Start 
2 x rj Location Designation command. 

* 

The option field defines the following options for each 
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command defined in Lhe command field. Tn the case of Replay 
command, replay speed (normal or slow) and current time stamp 
arc defined in the option field. In the case of Pause command, 
current time stamp is defined. In the case of Fast-Forward 
b command, fast-forward speed with replaying or not and current 
time stamp are defined. In the case of Reverse command, reverse 
speed wiLh replaying or rioL and current. Lime stamp are defined. 
In the case of Thumbnail-based Replay Start Location Designation 
command, the designated time stamp is set in the option field. 
10 The command £ield and the option field as described above 

allow both real-time replay and time-shift replay. 
Controller of Transmitter 

As shown in Fig. 2, Lhe controller 104 of the transmitter 
10 implements a database 104a, a device status manager 104b, 
15 and a data selector 104c. The database 104a stores a record 
composed ol device inloriiidLiuu, transmission Hag, 
transmission mode, and time stamp for each receiver 12- The 
device status manager 104b receives an analytic result from 
the command analyzer 105 to store as a record Lhe device 
20 information identifying the receiver 12 that has transmitted 
a corresponding command and Lhe delivery information composed 
of transmission flag, transmission mode, and Lime sLamp. The 
.device information identifies the receiver 12, which may be 
IP (Internet Protocol) address. The transmission mode includes 
25 "real -time", "t 1 me-shif t " and "not def i neri" . The transmission 
flag is one of TRUE and FALSE. The device information arid Lhe 
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time stamp arc extracted from a received command by the command 
analyzer 10b. 

The data selector 104c seloeLs one of the 
real-time multimedia data received from the data processing 
h section 102 and the time-shifted multimedia data received C corn 
the storage device 103 depending on the device information and 
the delivery information stored in the database 104a. 

in the case where a start" request command tor- 
real -time multimedia delivery has been received from the 

10 receiver 12, Lhe device status manager 104b searches the database 
104a for. the device information included in Lhe rece.i ved start 
request command. In this case, since no device information 
identifying Lhe receiver 12 is found, the device status manager 
104b registers a new record for the receiver 12 into the database 

15 104a. For example, the device information is set to the IP 
address ol Lhe receiver 12, Lhe transmission flag is set to 
TRUE, the transmission mode is seL to "real-time", and the time 
stamp is set to 1000. 

The data selector 104c monitors the database 104a at all 
20 times and, when no change in device status of the receiver 12 
occurs in the database 104a, selects the real-time multimedia 
data received from the data processing section 102 to output 
it to the transmission data processor 106. 

Thereafter, when a time-shift transi tion command has been 
25 received from the receiver 12, the device status manager 104b 
searches Lhe database 104a for the device information included 
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in the received time-shift transition command. In this case, 
since the device information identifying the receiver 12 is 
found, the device status manager 104b sets the transmission mode 
ot the record corresponding to the receiver 12 to "time-shift" 
b and similarly sets the time stamp thereof Lo the yhii'Led time 
stamp. When the transmission mode has been changed in Lho 
database 104a, Lho data selector 104c detects such a change 
and therefore selects the time-shit Led multimedia data received 
from the storage device 103 to output it to the transmission 
10 data processor 106. 

When a termination command has been received from the 
receiver 12 r thedevice status manager 104bscarchcs the database 
104a lor the device inlouua I Lon included in the received 
termination command. In this ease, since Lhc device information 
15 identifying the receiver 12 is found, the device status manager 
10 4b ye Is Llie transmission flag of the record corresponding 
to Lhc receiver 12 to "FALSE". When the data selector 104c 
detects such a change, the data selector 104c stops delivery 
of real-time or time-shifted multimedia stream. 
20 ! -t should be noted that the controller 104 may be composed 

of a program-controlled processor implementing the data 
selector 104c and the device aLaLufl manager 104b and a memory 
(not shown) separately implementing the database 104a. 

Detailed control operations of the transmitter 10 and 
25 the receiver 12 will be described hereafter. 

Operation of Transmic ccr 
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Referring to fig, J, the controller 104 first performs 
initial i zati on of f hp. database 1 04a such that the transmission 
flag is set to "FALSE" and the transmission mode to "real-time" 
(step Al) . Thereafter, it is determined whether the command 

5 receiver 108 receives a command packet from a receiver 12 through 

Lho unieast receiver 113 (step A2) . When no command has been 
received (No in step A2) , it is further determined whether the 
transmission Hag oi Lhe database 104a is "TRUE" (step A18) . 
When Lho transmission flag isnoL "TRUE" (No instep Al 8) , it means 
10 that neither real-time nor time-shifted multimedia data is 
delivered and therefore control goes back to the step A2 . 

When the transmission flag is "TRUE" (Yes in step A18), 
it means that real -time/ti me-shi f ted multimedia data is 
del i vered Lo the receiver 1.2 ident if iedby the devi ce information 
15 corresponding to the transmission flag of TRUE and therefore 
the control goes to step A12, which will be described Idler. 

When a command has been received {Yes in step A2), the 
command analyzer 105 analyzes the received command and the 
controller 104 performs delivery control depending on the 
20 command analytic result (steps A3-A11) . 

More specifically, when the received command is a start 
request command (Yes in step A3), the command analyzer I0 r 3 
instructs the command response section 107 to acknowledge 
receipt of the start .request command (step A4) . The command 
25 response section 107 transmits a response to the start request 
command to the receiver 12 through the un least, transmitter 112. 
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Thereafter, the device status manager 104b of the controller 
101 registers device information of the receiver \7 and writes 
corresponding delivery information into the database 104a such 
that its transmission flag is set to "TRUE" (step A5) and itu 
5 transmission mode is set Lo "real-Lime" (step A6) . 

When the received command is not a start request command 
(No in step A3) but a termination request command (Yes in step 
A7), the command analyzer 10b instructs Lhe command response 
secLion 107 to acknowledge receipt of Lhc tormina tion request 
10 command (stepAl9) . The command response sect i on 1.07 transmi ts 
■a response Lo Lhe termination request, command Lo the receiver 
12 through the unicast transmitter 112 . Thereafter, the device 
status manager 104b of the controller 104 writes the 
corresponding delivery information into Lhe database 104a such 
15 that its transmission flag is reset to "FALSE" (step A20) and 
then the transmission process is terminated (step A21) . 

When Lhe received command is neither a start request 
command (No in stop A"3) nor a termination request command (No 
in stop A7) but a time-shift transition command (Yes in step 
20 A8), the command analyzer 105 instructs the command response 
secti on 1 07 to acknowl edge reoei pt of the t.ime-shi ft transi tion 
command (step A9) . The command response section 107 transmits 
a response to the time-shift transition command to the receiver 
12 through the unicast transmitter 112 . Thereafter, the device 
25 Status manager 104b of the controller 104 wriLCS corresponding 
delivery information into the database 104a such thuL iL3 
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transmission mode is set to " time -shift " (sLcp A10) and its 
time stamp is set: f.o its designated value (step All). Such 
a time shift transition command Includes a thumbnail -based 

replay start location designation command, a pause command, 

5 a reverse command, a slow-replay command, or a fast-forward 
command. Hereinafter, il is assumed that a time shift 
transition command is a thumbnail-based replay start location 
designation command. 

Subsequently, it is determined whether the current 

10 transmission mode of the corresponding receiver 12 indicates 
"real-time" (step A12). When it indicates "real-time" (Yes 
in step A12), the data selector 104c of the controller 104 
receives real-time multimedia data Irom the data processing 
section '1.02 (step A 1.4) and transfers it to the transmission 

lb data processor 106'. The transmission data processor 106 

assembles data packets from the real-time multimedia data with 
time stamps , and outputs them to the multicast transmitter 111 
of the transmission section "109. Tn this way, the 
real time multimedia data is transmitted to the receiver 12 

^0 (step A15) and then the control goes back to the step A2 . 

Tn the case where the start request command is received 
after live performance or conference has started, the controller 
104 preferably attaches preceding thumbnail pictures to the 
real-time multimedia data to be transmitted. 

- 5 When the current transmission mode of the corresponding 

receiver )2 indicates "time -shift" {No in step A12) , the 
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controller 10-1 checks the presence or absence of data stored 
in the storage device 103 (step A13). When no 
time-shifted multimedia data is left in the storage device 103 
(Yes in step A13), the data selector 104c of the controller 

f j 104 receives real- time multimedia data from Lhe data processing 

section 102 (step A14) and transfers it to Lhe transmission 
data processor .1,06. The transmission data processor 106 
assembles data packets from Lhe real-time multimedia daLa with 
time stamps, and outputs Lhem to the multicast transmitter 111 
10 oi the transmission section 109. In this way, the 

real-time multimedia data is transmitted to the receiver 12 
(step A15) and then Che conLrol goes back to the stap A2 . As 
described before, when the real-time multimedia data is 
transmitted after I i ve performance or conference has started, 
15 the controller 104 preferably attaches preceding thumbnail 
pictures to the real-time multimedia data to be transitu L Led . 

When time-shifted multimedia data is left in the storage 
device 103 (No In step A13) , the data selector 104c of the 
controller 104 receives time-shifted multimedia data from the 
20 storage device 103 based on a time stamp value set in Lhe database 
104a corresponding to the receiver 1? (stepAlfi) and transfers 
it to the transmission data processor 106. The transmission 
daLa processor 106 assembles unicast data packets from Lhe 
time-shifted multimedia data with time-shifted time stamps and 
?. r ) real-time Lime stamps, and outputs Lhem to the unicast 

transmitter 1 12 of the transmission section 109. in this way, 
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the time-shilLod multimedia data is transmi tted to the receiver 
12 (step AT 7) and then the control goes back to the step A2 . 

Assumi ng that a t i me-shif t transition command is a reverse 
command, at the step All, Lhc device status manager 104b ol 
5 the controller 104 writes a current time stamp and a reverse 
interval received Irom the receiver 1.?. into the database 104a. 
At the step A16, the data selector 104c of the controller 104 
fetches time-shifted multimedia data from the storage device 
103 based on time stamp values sequentially calculated from 
10 the current time stamp and the reverse interval and outputs 
them to the transmission dataprocessor 106 . More specifically, 
the time stamp values are cal culatedby sequentially subtracting 
the reverse interval irom the current time stamp. h'or; exampl e, 
assuming that the current time stamp is 100 and the reverse 
15 interval is b, the data selector 104b sequentially fetches 
time-shifted multimedia data each correspond i nq to time stamps 
of 100, 95, 90, and so on from the storage device 1.03 at regular 
intervals (e . g . 1 ) . 

Assuming that a time-shift transition command is a 
20 slow-rcplay command, at the step All, the device status manager 
101b of the controller 104 writes a current time stamp and a 
slow replay interval received from the receiver 12 into the 
database 104a. At the step A16, the data selector 104c of the 
controller 104 fetches time-shifted multimedia data from the 
?5 storage device 103 based on time stamp values calculated from 
the current time stamp and the slow-replay interval and outputs 



2003 08/25 MON 10:44 FAX 03 3288 3222 Katsuragi Patent -» DICKSTEIN&OSHINSKY 



@ 028/056 



FQb-610 23 

them to the transmission data processor 10 6 . More specif ica l.ly, 
the time stamp values are calculated by sequentially adding 
the slow-rcplay interval to tho current time stamp. For example, 
assuming that the current time stamp is 100 and the slow-replay 
5 interval ia 0.1, the data selector 104b sequentially letches 
Lime-shifted multimedia data each corresponding to time stamps 
of 100, 100. 1, 100,2 and so on from the storage device 103 al 
regular intervals (e.g. 1) . 

Assuming that a time-shift transition command is a 

10 fast-forward replay command, at the step All , the device 

status manager 104b of the control ler 104 writes a current time 
stamp and a fast-forward interval received from the receiver 
12 into the database 104a. At the step A16, the data selector 
104c oT Lhe controller 104 leLches Lime-shifted multimedia data 

lb from the storage device 103 based on time stamp values calculated 
from the current time stamp and the fast-forward interval and 

■ 

outputs them Lo Lhe transmission data processor 106. More 
specifically, the time stamp values are calculated by 
sequentially adding the fast-forward interval to the current 

^0 time stamp, for example, assuming that the current time stamp 
is 100 and Lhe fast-forward interval is R, the data selector 
104b sequentially letches time-shifted multimedia data each 
corresponding to time stamps of 100, 105, 110 and so on from 
the storage device 103 at regular intervals (e.g. 1). 

5 Assuming that a time-shift transition command is a pause 

command, at the step Ail, the device status manager 101b of 
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Lhc conLroller 104 writes a current Lime stamp into Lhc database 
101a . At the step Al 6, the data selector 104c of the control 1 er 
1 04 fetches time-shifted mu It lined 1 a data from the storage device 

103 based on a time stamp value calculated Iroin the current 

5 time stamp and outputs it to the transmission data processor 
106. For example, assuming that the current time stamp is 100, 
the data selector 104b sequentially fetches 
time-shifted multimedia data corresponding to the time stamp 
of 100 from the storage device 103 at regular intervals (e.g. 
10 1) . When a pause release command has been received together 
with a current time stamp from the receiver 12, the controller 

104 sequentially receives the following 

time-shifted multimedia data from the storage device 103 to 
transmit them to the receiver 12. 

1 5 Operation or Receiver 

Referring to Fig. 4, the controller 124 first performs 
initialization such that the reception flag is set to "FALSE" 
and the reception mode to "real -time" (step Bl) . Thereafter, 
the controller 124 determines whether a command transmission 

20 request is received from outside user (stopB2). When a command 
transmission request has been received (Yes in step B2), the 
control ler 124 instructs the command transmi tter 122 to transmit 
a requested command to the transmitter 10 through the untcast 
transmitter 110 of the transmission section 116 (step B3) . Such 

25 a requested command may be a start request command, a 

thumbnail-based replay start location designation command, a 
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pause command, a reverse command, a fast-forward command, a 
slow-replay command, a termination request command or the likes . 

Thereafter, it is determined whether Lhc received data 
ana l.yzer 121 receives a command response at the unicast receiver 
5 118 (step D4) . When no command response has been received (No 
in step B4), it Is further determined whether the reception 
flag is "TRUE" (sLcpfllB) . When the reception flag i s not "TRUE" 
(No in step B18), it means that neither real-time nor 
time-shifted multimedia data is received and therefore control 
10 goes back to the step B2- 

Whenthe reception Hag is "TRUE" (Yea insLepBl8) , it means 
that real-time/time-shifted multimedia data is received and 
therefore the control yues to step Bll, which wi 11 be described 
later . 

15 When a command response has been received (Yes in step 

R4) , the received ddla analyzer 121 analyzes the received command 
response to determined whether it is a response to the start 
request command (step B5) . When it is not a response to the 
start request command (No in step D5) r the control goes to a 

2 0 step B8- When it is a response to the start request command 
(Yes in step Bb), the controller 124 sets the reception flag 
Lo "TRUE" (step B6) and the reception mode Lo "real-time " (stop 
R7) . 

Thereafter, the received data analyzer \?A determines 
15 whether the received command response is a response to the 
terminacion request command (step BS) . when the received 
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command is a response Lo Lhc termination request command (Yes 
in step B8 ) , the controller 12 4 sets the reception Hag to W FALSK" 
(step B19) and terminates the process (step D20) . 

When the received command response is not a response to 
5 the termination request command (No in step B8) , the received 
data analyzer 121 determines whether the received command 
response is a response to the t ime-shif t transition command 
(step B9) > When it is the response to the time-shift transition 
command (Yes in step B9) , the controller 124 sets the 

10 reception mode to "time- shirt" (slop BIO) . 

Subsequently, it is determined whether the current 
reception mode indicates "real-time" (step BV1) . when it 
indicates "real-time" (Yea in step Bll), the data reception 
selector 123 selects real-time multimedia data received Irom 

15 the multicast receiver 117 of the reception section 115 (step 
B15) and transfers it to the recopLion ddld pxocessoc 125- 

The reception data processor 12b , as necessary, performs 
thumbnail creation (step B16) before back: to the step B2 . In 
the step BIG, the reception data processor 125 monitors the 

.20 amount oi received real- time multimedia data and, when the amount 
of reoei ved data per uni t time exceeds a predetermined threshold, 
creates thumbnail pictures from the received 
real-Lime multimedia daLa to store the thumbnail pictures with 
time stamps into the storage device 126. 

25 When the current reception mode indicates "time-shift/' 

(No in stepBll) , the data reception selector 123 gets a real -time 
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time stamp of real-Lime multimedia data and a time-shifted time 
stamp of the? fcime-shil Led multimedia data attached Lo the 
real-time multimedia data (step D12) . Thereafter, the data 
reception selector 123 determines whether a difference between 
5 the time-shifted time stamp and the real-time time a I amp is 
smaller than a predetermined threshold, to decide on whether 
a lime- ah if ted picture catches up Lhc real-time picture (step 
B13) . 

When a time-shifted picture catches up the real-time 
10 picture (Yes in slop B13) , the controller 124 sets the 

reception mode to "real-time" (step B14) . The data reception 
selector 123 selects real-time multimedia data received from 
the multicast receiver 117 el the reception section 115 (step 
Bib) and transfers it to the reception data processor 125- The 

15 reception data processor 125 performs thumbnail creation (step 
B16) before back Lo the step B2. When a time-shifted picture 
does not catch up the real-time picture (No in step B13), the. 
data reception .selector 123 selects time-shifted mulLimedia 
data received from the received data analyzer 121 (step 017} 

20 and then the control goes back to the step B2 . 

Taking a teleconference as an example, a presenter may 
use an electronic version of presentation documents to provide 
an explanation to conferees . In the tel econference application, 
there is seldom a case where an entire picture is changed during 

25; expl ana t ion of a si nglc page. Accordingly, there is frequently 
used a delivery method such that difference data between a 
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current frame and an immediately preceding frame or a 
several -second-old frame is delivered to reduce the load on 
the network as much as possible. In such a case, when breaking 
across pages, a picture is changed and thereby the amount of 
5 real-Lime multimedia data is increased as shown in Fig. 8, 
Therefore, the receiver 12 can easily detect page turning. At 
such page change Liming, it is possible to create thumbnail 
pictures from the real-time multimedia data, by which the 
conferees can obtain an electronic version of presentation 
10 documents as the thumbnail pictures and further use the time 
slamps of the thumbnail pictures as indexes L.o designate replay 
start locations of the real-time multimedia data. Accordingly, 
when the present invention is applied to the teleconference 
application, it is possible lor a conleree attending to the 
15 conference to check subjects on the agenda from the beginning 
of the conference. Further, when a conferee leaves the 
conference lor a while, the conferee can check subjects on the 
agenda when the conferee is absent. Furthermore, a conferee 
can look at the foregoing several pages of the presentation 
20 documents delivered during the conference. 
Seque.n I; la 1 Opera L Ions 

Sequential operations between the transmitter 10 and the 
receiver 12 will, be described with reference to Figs. 9-11. 
As shown in Fig. 9, when a sLart request command has been 
25 sent from the receiver 1 2 to the transmitter 10, the transmitter 
10 sends a response to Che start request command back lo the 
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receiver 12 and sets the transmission mode Lo NN real- Lime" . At 
the rfinfiiver 17, when having recfiiveri the response to the .start- 
request command, the reception mode is set to "real-time - " In 
the transmitter 10, the controller 104 receives 

5 real-time multimedia data from the data processing section 102 
and sends iL to the receiver 12 in multicast. 

As shown in Fig. 10, when a slow-replaying request command 
has been senL Irum Lhe receiver 12 to the transmitter 10, the 
transmitter 10 sends a response to the slow-replaying request 
10 command bank to the receiver 12 and sets the transmission mode 
to "time-shi ft". Tn the transmitter 10, the controller 104 
receives time-shifted multimedia data trom the storage device 
103 and sends it to the receiver 12 in unicasL. 

As shown in Fj.g . 11, when a termination request command 
lb has been sent from the receiver 12 to Lhe transmitter 10, the 
transmitter 10 sends a response to the termination requeat 
command back to the receiver 12 and terminates the transmission 
process . 
Advantages 

20 As described above, the real-Lime multimedia stream 

del i very system according to the f i rst embodiment of the present 
invention is provided with a multicast nnicast 
changeover mechanism for changing bcLweon real-time mul timedia 
data delivery and time-shifted multimedia data delivery, which 

25 allows Lime-shifted replay in real-time multimedia streaming 
del i very . 
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Further, when Lhc receiver creates thumbnail pictures 
from the received real-times multimedia data, time stamps 
attached to the thumbnail pictures are stored into the storage 
device. Accordingly, by sending a desired time stamp to the 
b transmitter, a desired replay start location of 

real time multimedia data can be easily designated* 

Referring to Fig. 12, a real-time multimedia stream 
del Ivory system according to a second embodiment of the present 
invention is composed of a transmit Lor 10 and a receiver 13, 

10 which are connectedby a network 11 . Needless to say, aplurality 
of receivers 13 may be connected to the network 11. lilocks 
similar to those previously described with reference to Fig. 
1 are denoted by the same reference numerals and detailed 
descriptions thereof will be omitted. 

15 As described in the iirst embodiment, the L remain i t te r 

10 delivers real-time multimedia data to the receiver (s) 13 
in multicast through the network 11. When a time-shit L 
replay i ng request occurs, the transmi tter 1 0 delivers requested 
time-shilLcd multimedia data in unicast through the network 

20 11 to the receiver 13 which originates that request. 

Accordi ng to the second embodiment r in addition, a storage 
device 130 of each receiver 13 is provided to store 
real-time multimedia data received from the transmitter 10 
in multicast, which can bo used for time-shift replaying under 

25 control of the controller 128. More specifically , a controller 
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128 ol the receiver 13 instructs a reception data processor 

129 to store the received real-time multimedia data into the 
storage devi.ee 1.30, Depending on user's instruction, the 
controller 128 can select one oi an inside replay mode and an 

5 outside replay mode. The outside replay mode is that time-shift 
replaying is performed at the transmitter 10 as described in 
the first embodiment . The inside replay mode is that time-shift 
replaying is performed at the receiver 13 by reading 
timc-shii Led multimedia data from the storage device 130, The 

10 time-shifted multimedia data ia designated by inputting a 

time-shift transition command similar to the first embodiment. 

In order to store the necessary amount of 
real-time multimedia data received Iroin the transmitter 10/ 
the storage device 130 needs sufficient capacity, which may 

15 be implemented by hard disk drive (HDD). Further, the reception 
data processor 129 has functions of storing real-time multimedia 
data into the storage device 130 and instructing the storage 
device 130 to output t i me-shi f ted mul ti media data to the output 
section 127 under the control of the controller 120. 

20 As described above, according to the second embodiment, 

the i nsi de and outsi de repl ay modes can be selected at a receiver . 
Since the inside replay mode al 1 ows ti me-shi f ti nq replay without 
transferring multimedia data through the network 11, the load 
on the network 11 can be dramatically reduced. 

25 The above-described functions of the transmitter 10 may 
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be implemented by transmitter control programs running on a 
program-controliedprocessor . Similarly, the above^descri bed 
luneLions of the receiver 12 or 13 may be implemented by receiver 
control programs running on a program-controlled processor, 
b As shown in tig. 13, the data processing section 102, 

the control 1 er 104, the command analyzer 105, transmission data 
processor 106, Lhc command response socLion 107, and the command 
receiver 108 may be implemented by transmission control programs 
stored in a memory 202 running on a program- controlled processor 

10 201. Similarly, the received data analyzer 121/ the command 
transmitter 122 , data reception selector 123, the controller 
124/128, and the reception data processor 125/129 may be 
implemented by reception control programs .stored Ln a memory 
204 running on a program-controlled processor 203. The 

15 operations of the transmitter 10 and the receiver 12 or 13 are 
si.rnLl.dL to those described with reference to Kigs. 3 and 4, 
and therefore details arc omitted. 



